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Executive  Summary 

Joint  Universal  Lessons  Learned  System  (JULLS) 
Command  and  Control  (C2)  Analysis 
Summer  1998 


Question 

In  the  Summer  of  1998,  Air  Force  Studies  and  Analyses  Agency  (AFSAA)  employed  two 
Air  Force  Cadets  to  evaluate  JULLS  as  a  C2  analysis  data  source.  According  to  AF/XOOT,  Lt 
Col  Medvetz,  a  majority  of  exercises  are  designed  to  train  and  enhance  C2.  Our  main  question 
was  “Are  lessons  learned  a  viable  source  for  C2  analysis  data?” 

Customers 

There  were  no  external  customers  for  this  analysis.  This  analysis  was  internally 
generated  by  AFSAA  as  an  attempt  to  mine  C2  data  from  a  joint  source.  During  and  after  these 
activities,  we  coordinated  with  the  Air  Force  and  Joint  offices  handling  lessons  learned  collection 
and  dissemination.  They  helped  us  identify  which  databases  to  use,  and  they  provided  advice  and 
guidance  as  we  learned  and  analyzed  the  JULLS  data.  They  were  extremely  supportive  of  our 
efforts  and  generously  shared  their  experiences  and  knowledge  of  the  system,  processes,  and  data. 
This  work  culminated  with  a  brief  to  AF/XOOT  in  December  1999.  AF/XOOT  informally 
requested  AFSAA  to  participate  and  contribute  to  Air  Force  meetings  in  the  spring  of  2000  to 
update  the  lessons  learned  software.  This  may  promote  data  collection  needed  for  analysis, 
particularly  trend  analysis. 

Scope/Limitations 

Air  Force  Center  for  Knowledge  Sharing  Lessons  Learned  (AFCKSLL)  under  AF/XOOT 
provided  a  sample  of  1300  records  from  4/96  to  12/97.  Joint  Center  for  Lessons  Learned  (JCLL) 
under  JCS/J7  provided  908  records  categorized  by  the  Universal  Joint  Task  List  (UJTL),  which 
were  a  subset  of  the  13,000  entries  of  the  JULLS  database  JM971 .  Cadet  Sakamoto  sorted  the 
AFCKSLL  database  by  Joint  Chiefs  of  Staff  (JCS)  organization  categories  and  by  C2  keywords 
(e.g.  Air  Operations  Center  [AOC],  Joint  Forces  Air  Component  Commander  [JFACC|^  Trend 
analysis  was  also  attempted  with  data  split  over  three  equal  time  periods.  Cadet  James  sorted  by 
UJTL  designated  C2  categories  (strategic,  operational,  and  tactical)  and  C2  overall.  Both  Cadets 
had  approximately  six  weeks  to  complete  their  analysis.  Each  of  them  produced  a  separate 
written  report  of  their  findings  and  recommendations. 

Results  and  Aftermath 

Over  35%  of  the  records  in  both  databases  related  to  C2.  In  the  current  system, 
the  user  must  read  through  the  narratives.  Data  availability  and  completeness  vary  quite  widely 
based  on  the  input  source(s).  Currently,  database  fields  are  not  designed  for  quantitative  analysis. 
In  order  to  be  able  to  use  JULLS  for  C2  analyses,  the  system  needs  to  be  structured  to  collect 
relevant  C2  data.  Adding  short  fields  such  as  YES/NO  and  multiple  choice  would  aid 
quantitative  analysis  of  the  lessons  learned. 


Final  Words 


JULLS  databases  today  do  not  lend  themselves  easily  to  analysis.  JULLS  databases  must 
be  populated  with  more  than  keywords  and  narrative  to  facilitate  quantitative  analysis  --  and  the 
entire  process  should  be  engineered  to  be  simple  and  automated.  Additionally,  with  quantitative 
analysis  in  place,  the  Air  Force  lessons  learned  processes  and  remedial  action  programs  can  be 
managed  more  coherently. 
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By  Cadets  Jachin  Sakamoto  and  Chad  James 
Analysis  lead  by  Corinne  Wallshein,  AFSAA/SAAB 


This  analysis  was  two  pronged,  by  different  Cadets,  under  the  direction  of 
Battle  Management,  Command  and  Control  Branch  in  the  Force  Analysis 
Division  of  Air  Force  Studies  and  Analyses  Agency  (AFSAA).  Cadet  Jachin 
Sakamoto  was  a  rising  fourth  class  member  at  the  U.S.  Air  Force  Academy, 
from  Hawaii,  Cadet  Chad  James  was  a  rising  fourth  class  ROTC  member  at 
Louisiana  State  University,  from  Kansas. 

The  analysis  director  was  Mrs.  Corinne  Wallshein,  AFSAA/SAAB.  An  earlier 
version  of  this  brief  was  reviewed  by  analysts  in  AFSAA,  AF/XOOT,  Joint 
Warfare  Center’s  Joint  Center  for  Lessons  Learned,  and  the  Air  Force’s 
Center  for  Knowledge  Sharing,  Lessons  Learned. 

The  following  personnel  were  responsible  for  this  analysis  brief: 


Mrs.  Corinne  Wallshein 
Cadet  Jachin  Sakamoto 
Cadet  Chad  James 


AFSAA/SAAB 

AFA 

ROTC,  Louisiana  State  University 


Purpose 


■  Examine  JULLS  databases  to  quantify  C2 
Lessons  Learned  (LL) 

■  Evaiuate  JULLS  as  C2  data  source  for 
anaiysts 

■  Examine  JULLS  for  trends  in  C2  lessons  learned 

■  Recommend  changes,  as  appropriate,  to  JULLS 
for  easier  quantitative  analysis 
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The  Cadets  were  asked  to  sort  and  count  the  lessons  learned  in  the  database  by  C2 
category.  They  were  then  requested  to  evaluate  the  databases  as  a  C2  data  source  for 
trend  analysis  and  other  C2  analyses 


CJCS  3150.25  establishes  reporting  policies  and  general  responsibilities  for  the 
management  of  the  Joint  After  Action  Reporting  System  (JAARS).  It  applies  to  the  Joint 
Staff,  combatant  commands,  Services,  Combat  Support  Agencies  (CSAs),  Defense 
agencies  responsive  to  the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  OSD.  Further,  it 
applies  to  both  the  operational  and  training  environments  of  the  Armed  Forces.  JAARS  is 
the  formal  process  for  the  collection  and  dissemination  of  observations,  lessons  learned, 
and  issues  generated  from  joint  operations  and  exercises.  All  operations  precipitated  by 
an  executive  order  or  deployment  order  from  the  National  Command  Authority  (NCA) 
require  a  complete  JAAR.  All  CJCS  exercises  in  the  CJCS  Joint  Training  Master 
Schedule  (CJCSN  3501)  or  the  Quarterly  Schedule  of  Significant  Military  Exercises 
(CJCSN  3502)  require  a  JAAR.  The  type  of  report  for  joint  training  events  is  determined 
by  the  combatant  commander  during  the  Joint  Annual  Training  Plan  submission.  There 
are  three  types  of  reports:  a  complete  JAAR;  an  abbreviated  JAAR;  and  a  cancellation 
report.  Lessons  learned  databases,  reporting  in  the  Joint  Universal  Lessons  Learned 
System  (JULLS)  begin  with  a  user  generated  JAAR  report.  A  JAAR  may  consist  of  a 
summary  and  multiple  lessons  learned,  issues,  key  observations  and  assessments. 

JAAR  components  are  entered  into  JULLS  databases. 


Background:  what  is  JULLS? 

■  JULLS 

■  A  stand  alone  program 

■  Facilitates  after-action  reporting  and  database 
management 

■  JULLS  goal 

■  Learn  from  and  avoid  mistakes  of  past  exercises 
and  operations 

■  Track  Remedial  Action  Project  (RAP)  items 

■  JULLS  types 

■  Lessons  learned  include  issues  and  observations 

■  Summary  (mandatory) 

■  Assessment  (optional) 
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JULLS  was  congressionally  mandated  so  the  military  could  learn  from  past  mistakes  and, 
hopefully,  not  repeat  them.  The  Joint  Chiefs  of  Staff  J7  office  is  responsible  for  managing  all 
the  lessons  learned,  and  to  collect,  analyze,  distribute,  and  preserve  JULLS,  they  set  up  JCLL 
at  JWFC.  The  Air  Force  took  a  centralized  control  /  decentralized  execution  theme  in 
managing  JULLS.  AF/XOOT  oversees  AFCKSLL  which  was  established  in  October  1 997  at 
Langley  Air  Force  Base  (AFB),  Virginia.  AFCKSLL’s  vision  is  to  create  an  environment  within 
the  Air  Force  where  knowledge  sharing  is  a  “normal”  practice  at  all  levels  of  command.  And, 
in  order  to  share  knowledge,  we  first  have  to  acquire  it.  More  data  does  not  translate  into 
more  knowledge  directly.  There’s  a  process  to  convert  data  into  information,  and  then  into 
knowledge.  This  process  requires  tremendous  manual  effort  and  diligence  since  LL  are 
primarily  narrative  and,  usually,  highly  subjective.  None  of  the  LL  data  is  cataloged  or 
categorized  other  than  by  type  of  LL  and  level  of  security  classification.  Data  collection 
contributes  to  the  problem  of  too  much  data  and  too  little  knowledge.  Analysis  is  done  by 
reading  the  narratives  and  mentally  evaluating  the  LL,  then  conveying  those  findings  verbally 
and/or  in  writing  to  others. 

There  are  three  types  of  JULLS.  (1)  The  LL  includes  lessons  learned,  issues,  and 
observations.  It  records  a  specific  lesson  learned  or  an  issue  noted.  It  is  the  most  common 
type  of  JULL.  It  contains  one  or  more  UJTL  Task(s)  linked  to  the  LL.  (2)  The  summary  JULL 
is  mandatory.  It  provides  an  overall  picture  of  the  exercise  of  operation  objectives  and 
documents  the  facts  of  the  event.  It  contains  the  Joint  Mission  Essential  Task  List  (JMETL) 
tasks  as  one  or  more  of  the  UJTL  task.  (3)  The  Assessment  JULL  is  optional.  It  reports  the 
degree  of  success  obtained.  It  is  usually  from  CC,  staff,  evaluators,  or  trainers.  It  is  a 
complete  appraisal  of  each  objective  and  each  UJTL  based  JMET. 


Background:  Who's  who? 

■  JCS/J7  established  Joint  Warfighting  Center’s 
Joint  Center  for  LL  (JCLL)  at  Fort  Monroe 

■  AF/XOOT  established  Air  Force  Center  for 
Knowledge  Sharing  LL  (AFCKSLL)  at  Langley  AFB 

■  Other  Services  have  LL  systems 

■  Center  for  Army  LL  at  Fort  Leavenworth 

■  Navy  LL  System  at  Naval  Warfare  Development 
Command  Doctine  Development  Division  at  Newport 

■  Marine  Corp  LL  System  at  Marine  Warfighting 
Development  Integration  Division  of  MCCDC  at  Quantico 


...to  shed  light 


These  agencies  are  responsible  for  collecting,  processing,  exploiting,  and  disseminating  LL.  With 
the  implosion  of  web  based  technology,  there’s  lots  of  LL  on  the  Internet  and  SIPRNET,  available 
for  personnel  to  leaf  through  with  key  word  searches. 

Collecting  Lessons  Learned  (LL)  is  a  requirement  for  all  Services.  There  are  various  databases  of 
LL  on  web  sites  easily  accessible  to  the  defense  community.  A  sampling  is  below: 

ACC  Center  for  Lessons  Learned  --  http://157.214.212.52/samples/search/query.htm 
ACC  Lessons  Learned  Categories  --  http://157.214.212.52/acc/acc_toc.htm 
AFCKSLL  Internet  Database  -  http://1 31 .6.21 6.21 
AFCKSLL  SIPRNET  Database  --  http://157.214.212.51 
Joint  Staff  JULLS  Database  is  at: 

http://www.eucom.smil.mil/eccs-as/info_center/lessons/isrch/advanced_julls.html 
EUCOM  Center  for  Lessons  Learned  Command  Data  Base  Search  Engine  is  at: 
http://www.eucom.smil.mil:80/eccs-as/info_center/lessons/isrch/simple_eujulls.html 


There  are  many  JULLS  databases.  Each  CINC  has  a  command  database  for  internal  use.  There 
were  three  joint  staff  level  JULLS  databases.  Two  resided  at  JCLL:  the  archived  and  the  current 
database.  The  third  database  is  the  Remedial  Action  Program  (RAP)  database  at  JCS/J7’s 
Evaluation  and  Analysis  Division  (EAD).  Both  JCLL  and  AFCKSLL  had  several  data  bases.  We 
chose  one  from  each  organization:  the  AFCKSLL  was  the  basis  for  Cadet  Sakamoto’s  analysis 
and  the  JCLL  database  was  the  basis  for  Cadet  James.  Our  efforts  were  geared  to  examining 
JULLS  for  C2  lessons  learned.  This  was  done  by  first  selecting  the  database,  reading  the  narrative 
entries,  sorting  the  entries  into  categories  of  C2  and/or  exercises,  attempting  trend  analysis  on  the 
JULLS,  and  then  suggesting  what  might  be  done  to  JULLS  to  make  it  easier  to  analyze. 


Background:  AF  LL  status 


■  Ad  hoc  LL  exploitation 

■  No  standard  impact  on  AF  performance  reviews 

■  GAO  Report,  Aug  95,  Military  Training:  Potential 
to  Use  LL  to  Avoid  Past  Mistakes  is  Largeiy 
Untapped  true  in  1998  according  to  AFCKSLL 

■  AF  web  based  LL  collection  and  dissemination 

■  Full  text  search  or  keyword  search  possible 

■  Challenge  to  inject  past  LL  into  planned  exercises  and 
operations 

■  AF  member  of  CJCS  Remedial  Action  Program 

■  Only  Joint  Staff  JULLS  corrective  action  program 

■  Formal  Service  requirement  in  CJCSI  3150.01 
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Currently,  in  the  AF,  LL  exploitation  is  labor  intensive  and  ad  hoc.  Whether  you  do  a  thorough  job  of  filling  out  JAARS 
or  not,  whether  you  read  through  past  LL  or  not,  and  whether  you  are  aware  of  LL  system  or  not  has  no  standard 
impact  on  your  individual  or  unit  performance  review(s). 

1995  GAO  findings  were:  Marine  Corps,  Air  Force,  and  Navy  do  not:  (1)  include  all  significant  information  from 
training  exercises  and  operations  in  their  LL  programs;  and  (2)  analyze  their  LL  information  to  identify  trends  in 
performance  weaknesses.  For  the  most  part,  the  dissemination  of  LL  information  is  adequate.  However,  the  Air 
Force  does  not  make  this  information  readily  available  to  all  potential  users.  AF’s  decentralized  LL  databases  are 
maintained  at  each  major  command’s  headquarters  and  therefore  are  not  easily  accessible  to  units  throughout  the  AF. 
GAO  recommends  that  the  Secretary  of  Defense  direct  the  Secretaries  of  the  Air  Force  and  the  Navy,  the  Chairman  of 
the  Joint  Chiefs  of  Staff,  and  the  regional  commands  in  chief,  as  appropriate,  to  (1)  establish  controls  to  ensure  that  all 
significant  LL  information  collected  from  combat  training  centers,  fleet  exercises,  and  other  major  training  exercises  are 
recorded  in  LL  databases;  (2)  analyze  LL  information  so  that  trend  data  can  be  developed  to  identify  recurring 
problems;  (3)  provide  training  to  key  personnel  in  the  use  of  LL  information;  and  (4)  incorporate  effective  validation 
procedures,  such  as  testing  corrective  actions  in  joint  training  exercises,  into  LL  programs.  AF’s  LL  program  is  the  only 
one  of  the  services’  programs  that  is  decentralized.  As  a  result,  each  of  the  AF’s  six  major  operational  commands  is 
responsible  for  developing  and  managing  its  own  LL  program.  AF  regulations  do  not  require  that  the  commands’ 
programs  be  uniform,  so  each  command  can  take  different  approaches  to  operating  its  program.  The  programs  were 
designed  to  account  for,  act  on,  and  share  LL  information  throughout  the  command,  but  not  throughout  the  AF.  The  AF 
also  operates  a  limited  LL  program  at  its  headquarters.  This  program,  staffed  by  two  people,  addresses  LL  information 
that  results  only  from  the  AF’s  participation  in  joint  exercises  or  operations  or  that  affects  more  than  one  of  the  major 
commands’  missions.  AF  LL  managers  at  one  major  command  were  not  successful  in  obtaining  after-action  reports 
covering  units’  participation  in  various  exercises  and  operations.  Conversely,  another  AF  major  command  had 
established  a  process  to  monitor  the  submission  of  after-action  reports  by  subordinate  units.  A  1993  AF  LL  report  on 
Operations  Restore  Hope  stated  that:  “Almost  every  problem  occurring  during  Operation  Restore  Hope  has  already 
been  documented  in  JULLS  as  a  result  of  previous  exercises  and  contingencies.  There  appears  to  be  a  continuing 
trend  of  failure  to  fix  problems  already  know  [sic]  to  exist.  We  end  up  paying  again  to  achieve  the  same  undesirable 
results.”  Lessons  are  not  routinely  analyzed  to  identify  recurring  deficiencies. 

AFCKSLL  consolidating  AF  inputs  on  world  wide  web  and  SIPRNET.  Searches  on  text  and  key  words  is  simple, 
whereas  analysis  requires  considerably  more  effort.  CJCSI  3150.01  describes  CJCS  Remedial  Action  Program  and 
spells  out  service  requirements  to  designate  coordinators  at  the  action  officer  and  planner  levels.  AF/XOOT  has  this 
responsibility. 


Methodology 


■  Examine  databases  for  C2  LL 

■  AFCKSLL  provided  1300  records  from  4/96  to  12/97 

■  Sorted  by  JCS  organizational  categories  (J1-  J8) 

■  Sorted  by  C2  keywords  (e.g.  AOC,  JFACC,  ATO,  Exercises) 

■  Trend  analysis  attempted  on  time  sequenced  segments 

■  JCLL  provided  908  records  validated  by  the  CINCs 

■  Considered  true  lessons  learned  dating  from  1991  and 
including  Operation  Just  Cause 

■  Data  categorized  by  Universal  Joint  Task  List  (UJTL) 

■  Sorted  by  C2  strategic,  operational  and  tactical  categories 

■  Analyzed  proportion  of  C2  in  UJTL  categories 


AFCKSLL  focused  on  AF;  JCLL  focused  on  joint 


Databases  contain  a  great  deal  of  narrative,  subjective  information.  The  quality  and  usefulness  of 
this  information  ranges  from  good  to  not  so  good.  A  considerable  part  of  the  work  done  by 
AFCKSLL  and  JCLL  for  their  clients  involves  qualitative,  subjective  analysis  and  synthesis  using 
previous  military  experience  and  professional  judgment. 


Cadet  Sakamoto  used  a  JULLS  database  from  AFCKSLL  containing  approximately  1283  records 
and  spanning  a  time  period  from  25  April  1996  to  5  December  1997.  While  the  entire  JULLS 
database  programs  has  over  20,000  records,  this  analysis  utilized  a  sample  database  for  two 
reasons:  (1 )  A  smaller  sample  was  easier  and  faster  to  query;  (2)  Additional  records  were  difficult 
to  obtain  and  import  into  Microsoft  Access.  Several  attempts  were  made  to  provide  key  word 
search  query  capabilities  on  the  world  wide  web  and  SIPRNET  by  organizations  such  as 
USEUCOM  and  ACC.  Most  of  the  INTELINK  sites,  which  allow  actual  downloading  of  JULLS 
databases  only  provide  the  ASCII/text  format,  were  difficult  to  manipulate.  Other  sites  only 
allowed  viewing  on  a  file  by  file  basis. 


Cadet  James  used  a  JULLS  database  from  JCLL  containing  908  records,  810  of  which  were 
linked  to  one  or  more  task  in  the  UJTL.  They  were  a  subset  of  over  13,000  entries  in  a  JULLS 
database  called  JM971 .  Records  consisted  of  summaries,  assessments,  lessons  learned,  issues 
and  key  observations.  In  the  JCLL  Bulletin,  CCTI  96-3,  on  C2  of  Joint  Air  Operations,  JCLL 
retrieved  the  following  UJTLs:  OP  1 .5.3  --  Gain  and  maintain  air  superiority  in  theater  of 
operations/JOA;  OP  3.1.2  ~  Assign  Joint/Multinational  operational  firepower  resources;  OP  6.1 .3  - 
-  Provide  aerospace  control.  They  discussed  three  topics:  (1)  Implementation  of  a  JFACC;  (2) 
Production  of  the  Air  Tasking  Order;  (3)  Component  coordination.  They  did  not  use  Section  5, 
titled  Command  and  Control,  in  the  UJTL.  Cadet  James  concentrated  solely  on  Section  5. 


AF  Database  Distribution 


Keyword  sort:  just  under  50%  of  reports  ‘hit’ 

Database  Distribution  by  Topics 
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A  topical  approach  was  used  here  to  address: 

•  Procedures  such  as  the  ATO  and  training 

•  Organizations  and  Offices  such  as  the  AF,  Army,  Navy,  Marines,  AOC  and  JFACC 

•  Exercises  such  as  Roving  Sands,  Blue  Flag  and  Red  Flag. 


This  approach  seemed  to  be  exclusive.  It  failed  to  include  a  majority  of  the  records,  though  this 
could  be  handled  by  increasing  the  number  of  categories. 


CATEGORY 

SIZE 

PERCENTAGE 

KEY  WORDS  USED 

View  AOC 

58 

4.52% 

AOC,  Air  Operations  Center 

View  Air  Force 

103 

8.03% 

Air  Force,  USAF 

View  Army 

83 

6.47% 

Army 

View  Navy 

32 

2.50% 

Navy,  USN 

View  Marine  Corps 

16 

1.25% 

Marine,  USMC 

View  Blue  Flag 

18 

1 .40% 

Blue  Flag 

View  Red  Flag 

3 

0.23% 

Red  Flag 

View  JFACC 

35 

2.73% 

JFACC,  Joint  Force  Air 

View  JSTARS 

7 

0.55% 

STARS,  eagle,  Joint  Surveillance  Target  Attack  System 

View  Roving  Sands 

112 

8.74% 

Roving  Sand 

View  ATO 

186 

14.51% 

tasking  order,  ATO 

View  Training 

443 

34.56% 

training,  exercise,  simulation. 

Records  Not  Included 

649 

50.6% 

scenario 

AF  overlap  in  JCS  categories 

■  J6  (35%),  J3  (25%),  &  J1  (35%)  higher  than  others 

■  95%  of  records  in  J6  mentioned  communication,  coordination,  or  both 

■  Lots  of  double  counting:  146%of  data  shown 


Database  Distribution  by  J1-J8 
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40.00% 

35.00% 

30.00% 

25.00% 

20.00% 

15.00% 

10.00% 


The  database  was  sub-categorized  by  the  different  Joint  Staff  components,  J1  through  J8.  The 
reasoning  behind  this  approach  was  each  of  the  Joint  Staff  components  could  pick  out  problems 
that  pertain  to  their  field.  Although  this  approach  may  uncover  major  issues  to  be  addressed  at  the 
strategic  level,  many  of  the  issues  and  recommendations  in  the  AF  JULLS  database  were  directed 
at  the  operational  level.  Each  sub-categorization  was  limited  to  a  keyword  search.  The  entire  text 
portion  of  each  record  was  searched  and  depended  upon  the  word  used  by  personnel  in  the  data 
entry  stage  of  the  database.  The  following  key  word  list  was  generated  by  Cadet  Sakamoto  after 
he  had  read  through  a  sample  of  the  database  to  glean  an  understanding  of  the  common 
terminology  in  the  records.  As  shown,  J1 ,  J3  and  J6  have  the  most  entries.  One  could  infer  that 
J1 ,  J3,  and  J6  have  the  most  problems. 


CATEGORY  SIZE 

PERCENT 

KEY  WORDS  USED 

J1  -  Manning 

452 

35.26% 

Manning,  Personnel,  manpower,  deployed,  manned 

J2  -  Intel 

226 

17.63% 

Intel,  intelligence,  data,  surveillance,  tracking,  targeting 

J3  -  Operations 

338 

26.37% 

operations,  ops 

J4  -  Logistics 

96 

7.49% 

Logistics,  log 

J5  -  Strategy 

34 

2.65% 

strategy,  doctrine,  policy,  regulation 

J6-C2 

63 

4.91% 

C2,  C3,  C4,  command  and  control 

J6  -  Communication 

392 

30.58% 

Communication,  phone,  radio,  interface,  computer,  net,  link 

J7  -  Coordination 

166 

12.95% 

coordination,  aligned,  synchronization,  conflicting, 
different  databases,  integration,  interoperability 

J8  -  Assessment 

102 

7.96% 

structure,  resources,  assessment 

Records  Not  Included  1 

0.08% 

TOTAL:  1869  out  of  1282  145.79% 


An  attempt  was  made  to  analyze  the  communications,  coordination,  and  C2  categories  by  time  to 
discover  recognizable  trends.  The  goal  behind  this  analysis  was  to  show  that  the  quantity  of 
problems  in  certain  areas  decreased  over  time  --  suggesting  that  the  military  was  learning  from  its 
mistakes.  The  data  was  divided  into  three  approximate  six  month  sections,  the  last  section  being 
slightly  larger  than  the  other  two.  The  time  frames  were:  4/26/96  to  10/26/96;  10/26/96  to  4/26/97; 
and  4/26/97  to  12/5/97.  The  bar  charts  show  actual  record  counts  and  percentages  in  relation  to 
the  number  of  records  contained  in  each  period.  While  these  results  are  inconclusive  and  no 
trends  are  evident,  additional  data  covering  a  larger  time  frame  may  be  more  revealing.  Data 
entries  come  in  spikes  following  exercises  and  operations.  Larger  time  sections  would  allow  the 
data  in  each  section  to  be  more  normalized  in  relation  to  other  sections. 


CATEGORY 

NUMBER 

PERCENTAGE 

4/96-12/96 

12/96-5/97 

4/97-12/97 

4/96-12/96 

12/96-5/97 

4/97-12/97 

C2 

1 

35 

23 

0.32% 

8.10% 

4.33% 

Communication 

72 

144 

176 

22.71% 

33.33% 

33.15% 

Coordination 

4 

94 

68 

1.26% 

21.76% 

12.81% 

Pure  C2 

1 

16 

10 

0.32% 

3.70% 

1.88% 

Pure  Communication 

71 

100 

130 

22.40% 

23.15% 

24.48% 

Pure  Coordination 

3 

55 

29 

0.95% 

12.73% 

5.46% 

Overlap  C2  &  Comm 

0 

15 

12 

0% 

3.47% 

2.26% 

Overlap  Comm  &  Coord 

1 

35 

38 

0.32% 

8.10% 

7.16% 

Overlap  Coord  &  C2 

0 

10 

5 

0% 

2.31% 

0.94% 

Overlap  C2,  Comm  &  Coord 

0 

6 

4 

0% 

1.39% 

0.75% 

Joint  database  C2  distribution 

42%  re  le  to  C2 


■  C2 

■  Other 


344  C2  related  records  out  of  810  categorized  by  UJTL,  out  of  908  total 
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...to  shed  light 


2A10/DO 


When  Cadet  James  sorted  UJTL  Section  5  Command  and  Control  records 
out  of  the  total,  he  counted  344  records.  344/810  is  42.47%  Therefore,  at 
least  42%  of  the  records  in  the  database  relate  to  C2. 


In  Cadet  Sakamoto’s  Joint  Staff  categorization,  where  J6  had  the  key  words 
of  C2,  C3,  C4,  command  and  control,  communication,  phone,  radio, 
interface,  computer,  net,  and  link,  there  were  455  records  out  of  1 282. 

Using  the  key  words  of  C2,  C3,  C4,  and  command  and  control  for  J6’s  C2, 
there  were  63  records.  Using  the  key  words  of  communication,  phone, 
radio,  interface,  computer,  net,  and  link  for  J6’s  communication,  there  were 
392  records.  Total  J6  categorization  came  to  455  records.  Though  there 
was  undoubtedly  double  counting,  the  total  percentage  of  455/1282  is 
35.49%. 


Joint  UJTL  Break-down 


UJTL  Tasks  by  Level  of  War 


Four  C2  Categories 


strategic  - 
Theater 
28% 

Strategic 

National 

16% 


Tactical 

2% 


Operational 

54% 


UJTL  categories  5.1 , 5.4,  5.5,  5.7,  and  5.8  only 


2/10/00 


The  UJTL  records  were  sorted  into  several  section  5  categories,  namely: 

•  Section  5.1  -  Operate  and  Manage  Communications  and  Information  at  the  appropriate 
level  (There  are  four  levels:  SN  is  Strategic  National;  ST  is  Strategic  Theater;  OP  is 
Operational;  and  TA  is  Tactical) 

•  Section  5.4  --  Provide  Direction  to  Forces  at  appropriate  level 

•  Section  5.5  --  Organize  Joint  Headquarters 

•  Section  5.7  --  Coordinate  and  Integrate  Joint/Multinational  Interagency  Support 

•  Section  5.8  --  Provide  Public  Affairs  in  Theater  of  Operations/Joint  Operations  Area  (JOA) 

The  309  records  broke  out  into  the  following  levels: 


SN 

“ 

49 

ST 

“ 

87 

OP 

— 

167 

TA 

— 

6 

These  numbers  translate  into  the  percentages  shown  in  the  pie  chart. 


■  Operational 

■  OP  5.1  22% 

■  OP  5.4  12% 

■  OP  5.5  32% 

■  OP  5.7  12% 


Joint  C2  Levels  by  UJTL 


strategic 

■  National  Level 

■  SN  5.1 

■  SN  5.4 

■  Theater  Level 

■  ST  5.1 

■  ST  5.4 


50% 

38% 

35% 

39% 


Percentage  of  Operational  Lessons  Learned  shift  from  Strategic  Levels 
4’  Operate/Manage  Communications  and  Info  at  Appropriate  Level  (5.1) 
4  Provide  Direction  to  Forces  at  Appropriate  Level  (5.4) 

Organize  Joint  Headquarters  (5.5) 

Coordinate  &  Integrate  Joint/Multinational  Interagency  Support  (5.7) 


...to  shed  tight 


2/10A)0 


This  slide  provides  a  snapshot  of  the  total  distribution  of  309  C2  related 
records,  shown  in  the  center  pie  chart,  together  with  the  percentages  in 
the  three  UJTL  levels  of  warfare:  Strategic  National  (SN),  Strategic 
Theater  (ST),  and  Operational  (OP).  The  box  at  the  bottom  of  the  slide 
summarizes  the  difference  between  the  warfare  levels. 


The  percentage  of  LL  linked  to  OP  5.1 ,  Operate  and  Manage 
Communications  and  Information  at  the  Operational  Level,  is  less  than 
that  SN  5.1  and  ST  5.1 ,  Operate  and  Manage  Communications  and 
Information  at  the  Strategic  Level  The  percentage  of  LL  linked  to  OP 
5.4,  Provide  Direction  to  Forces  at  the  Operational  Level  is  less  than 
SN  5.4  and  ST  5.4,  Provide  Direction  to  Forces  at  the  Strategic  Level 


The  percentage  of  LL  linked  to  OP  5.5,  Organize  Joint  Headquarters, 
and  OP  5.7,  Coordinate  and  Integrate  Joint/Multinational  Interagency 
Support,  is  much  greater  than  similarly  numbered  UJTL  strategic  tasks. 


UJTL  Strategic  National  C2 


■  16%  related  to  Strategic  National 


Strategic  Level  -  National  Military  Tasks 


Other 

13% 


Provide 
Strategic 
Direction 
to  Forces 
Worldwide 


SN  5,1 
50% 


Operate  / 
Manage 
Global 
Strategic 
Comm  & 
Info  Sys 


Bmi 


The  records  sorted  out  as  follows: 


UJTL 

Description 

Number 

Percentage 

SN  5.1 

Operate/Manage 
Global  Strategic 

Communications 

and  Information 

System 

28 

50% 

SN  5.4 

Provide  Strategic 

Direction  to  Forces 

Worldwide 

21 

37.5% 

Other 

7 

12.5% 

TOTAL 

56 

100% 

All  C2  records  totaled  344,  so  the  percentage  related  to  Strategic 
National  is  56/344  =  1 6%. 


UJTL  Strategic  Theater  C2 

■  28%  related  to  Strategic  Theater 
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Strategic  Level  -  Theater  Tasks 


Other 

10% 


16%  Prepare  &  Plan  Orders 


...to  shed  light 


2/10/00 


UJTL 

Description 

Number 

Percentage 

ST  5.1 

Operate/Manage  Theater 

Communications  and 

Information  Systems 

34 

35% 

ST  5.3 

Prepare  and  Plan  Orders 

15 

16% 

ST  5.4 

Provide  Strategic  Direction 

38 

39% 

Other 

10 

10% 

TOTAL 

97 

100% 

94/344  =  28% 


UJTL  Operational  C2 


■  54%  related  to  Operational  C2 


Operational  Level  Tasks 


Provide 
Public 
Affairs  in 
Theater  of 
Operations  / 
JOA 


Acquire  & 
Communicate 
Operational 
Level  Info  & 
Maintain  Status 


UJTL 

Description 

Number 

Percentage 

OP  5.1 

Acquire  and  Communicate  Operational 
Level  Information  and  Maintain  Status 

40 

22% 

OP  5.4 

Command  Subordinate  Operational 

Forces 

22 

12% 

OP  5.5 

Organize  Joint  Forces  Headquarters 

60 

32% 

OP  5.7 

Coordinate  and  Integrate  Joint/ 
Multinational  and  Interagency  Support 

22 

12% 

OP  5.8 

Provide  Public  Affairs  in  Theater  of 

Operations/JOA 

17 

9% 

Other 

24 

13% 

TOTAL 

185 

100% 

185/344  =  54% 


Findings 


■  35  -  42%  of  JULLS  relate  to  C2 

■  In  current  system,  user  must  read  through 
narratives  to  get  information  and  knowledge 

■  Varied  data  availability  and  completeness 

■  Only  required  short  field  is  classification 

■  Importing  into  ACCESS  database  took  time  &  effort 

■  Work  in  progress  to  make  JULLS  more  accessible 

■  Current  capability  to  do  analysis  highly  manpower  intensive 

■  Fields  In  database  weren’t  designed  for  quantitative  analysis 

We  can  rnd(B  I  easier  to  learn  these  less^ 


—to  shed  light 


Both  Cadets  found  that  approximately  40%  of  the  LL  related  to  C2. 


The  user  must  read  through  the  narratives  to  get  a  feeling  for  what’s  in  the  databases.  The  data 
availability  and  completeness  varies  by  records  within  the  same  database  and  between 
databases.  The  only  required  short  field  is  classification  level.  The  databases  from  JCLL  and 
AFCKSLL  used  specially  developed  software  to  view  the  records.  This  specially  developed 
software  did  not  allow  for  easy  import  into  ACCESS  which  complicated  the  Cadets’s  job  of 
counting,  sorting,  and  evaluating  the  data. 


When  we  spoke  with  JCLL  personnel,  we  learned  that  they  were  working  on  database  inputs  and 
formats  and  they  were  institutionalizing  the  process  of  sharing  LL  with  testing  and  exercise 
personnel.  Their  approach  was  to  grow  their  staff  to  read  through  and  summarize  the  pertinent 
data  to  present  to  the  testing  and  exercise  personnel.  I  have  a  copy  of  their  TRAIL  BLAZER 
report. 


At  AFCKSLL,  they  were  working  on  a  web-based  input  and  search  database.  They  wanted  to 
make  the  LL  easily  available  to  AF  personnel.  They  did  not  have  the  staff  and  did  not  have  plans 
to  significantly  grow  the  staff  to  internally  read,  process,  summarize,  and  disseminate  the 
summaries  of  AF  LL. 


The  way  to  make  it  easier  to  learn  lessons  is  to  change  the  input  process  so  the  retrieval  process 
is  less  manually  intensive.  It  would  make  quantifying  LL  easier,  it  would  make  it  easier  to 
convert  data  into  information  and  then  into  knowledge,  and  it  would  make  disseminating  the 
knowledge  easier. 


Recommendations 


■  Plan  to  use  JULLS  in  quantitative  analyses 


■  Restructure  JULLS 

■  Survey  defense  community  for  needed  analysis  info 

■  Change  collection  process  to  get  relevant  data 

■  Add  survey  fields  that  are  Y/N  and  multiple  choice 
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■  Expand  lessons  learned  sharing 


■  Communicate  with  AF 

■  Emphasize  importance  of  data  input 

■  Educate  AF  personnel 

■  Link  LL  processes  with  performance  reviews 

■  Report  widely  on  Remedial  Action  Program 

■  Gather  quantitative  C2  data  &  metrics  from  JULLS 

■  Share  data  and  metrics  with  AF 

■  Analyze  using  various  techniques,  including  trend  analysis 


...to  shed  light 


2/1  OAK) 


“...never  feel  badly  about  making  mistakes  as  long  as  you  take  the  trouble  to  learn  from  them. ..but 
it’s  not  just  learning  things  that’s  important.  It’s  learning  what  to  do  with  what  you  learn  and 
learning  why  you  learn  at  all  that  matters.”  The  Phantom  Tollbooth.  by  Norman  Juster,  1961 ,  page 
233. 


One  of  the  most  glaringly  obvious  recommendations  of  this  work  was  to  restructure  the  JULLS 
inputs.  If  there  were  more  quantifiable  fields  in  the  LL  databases,  then  JULLS  would  be  easier  to 
quantify.  This  corresponds  to  adding  Yes/No  (Y/N)  fields  and  multiple  choice  fields  based  on 
subjective  inputs. 


Another  recommendation  was  to  expand  LL  sharing  of  relevant  LL.  This  recommendation  includes 
the  work  it  will  require  from  our  senior  leaders  to  make  LL  relevant:  communicating  the  importance 
of  inputting  LL,  linking  the  quality  of  LL  input  to  individual  and  unit  performance  ratings,  and 
informing  the  AF  of  the  status  of  RAP  from  LL. 


Finally,  once  we  have  quantifiable  fields  in  the  database  records  and  collect  them,  then  we  have  a 
built  a  body  of  data  for  trend  analysis  and  other  analyses.  The  AF  will  be  empowered  to  achieve 
the  following:  “We  hope  to  share  with  the  community  the  lessons  we  have  learned  so  that  our 
successes  can  be  exploited  instead  of  relearned  and  so  our  failures  can  be  avoided  instead  of 
recommitted.”  Modeling  for  Campaign  Analysis:  Lessons  for  the  Next  Generation  of  Models. 
Executive  Summary,  by  Richard  J.  Hillestad,  Bart  Bennett,  and  Louis  Moore,  Rand  Corporation, 
1996,  page  5. 


Recommendations 


■  Plan  to  use  JULLS  in  quantitative  anaiyses 

■  Restructure  JULLS 

■  Survey  defense  community  for  needed  analysis  info 

■  Change  collection  process  to  get  relevant  data 

■  Add  survey  fields  that  are  Y/N  and  multiple  choice 

■  Expand  lessons  learned  sharing 

■  Communicate  with  AF 

■  Emphasize  importance  of  data  input 

■  Educate  AF  personnel 

■  Link  LL  processes  with  performance  reviews 

■  Report  widely  on  Remedial  Action  Program 

■  Gather  quantitative  C2  data  &  metrics  from  JULLS 

■  Share  data  and  metrics  with  AF 

■  Analyze  using  various  techniques,  including  trend  analysis 
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One  of  the  most  glaringly  obvious  recommendations  of  this  work  was  to  restructure 
the  JULLS  inputs.  If  there  were  more  quantifiable  fields  in  the  LL  databases,  then 
JULLS  would  be  easier  to  quantify.  This  corresponds  to  adding  Yes/No  (Y/N)  fields 
and  multiple  choice  fields  based  on  subjective  inputs. 


Another  recommendation  was  to  expand  LL  sharing  of  relevant  LL.  This 
recommendation  includes  the  work  it  will  require  from  our  senior  leaders  to  make  LL 
relevant:  communicating  the  importance  of  inputting  LL,  linking  the  quality  of  LL  input 
to  individual  and  unit  performance  ratings,  and  informing  the  AF  of  the  status  of  RAPs 
from  LL. 


Finally,  once  we  have  quantifiable  fields  in  the  database  records  and  collect  them, 
then  we  have  a  built  a  body  of  data  for  trend  analysis  and  other  analyses.  The  AF 
will  be  empowered  to  achieve  the  following:  “We  hope  to  share  with  the  community 
the  lessons  we  have  learned  so  that  our  successes  can  be  exploited  instead  of 
relearned  and  so  our  failures  can  be  avoided  instead  of  recommitted.”  Modeling  for 
Campaign  Analysis:  Lessons  for  the  Next  Generation  of  Models,  Executive 
Summary,  by  Richard  J.  Hillestad,  Bart  Bennett,  and  Louis  Moore,  Rand  Corporation, 
1996,  page  5. 


